[00:01.280 --> 00:05.120]  Ah, come back here. Why does that keep doing that?
[00:05.600 --> 00:09.560]  All right, great. We are here and ready to do securing in,
[00:09.560 --> 00:13.280]  path testing, great spaghetti monster, as I like to call it, or
[00:13.280 --> 00:18.400]  Kubernetes. This is who I am, Kat Fitzgerald.
[00:18.400 --> 00:22.840]  You can find me on Twitter at Rainbow Kat. I've been doing this a long time.
[00:22.840 --> 00:26.580]  I'm the CEO of B-Sides Chicago. I was COO for
[00:26.580 --> 00:30.820]  the Diana Initiative last year, part of B-Sides Pittsburgh,
[00:30.820 --> 00:34.900]  and also my first DEFCON was DEFCON 3.
[00:34.900 --> 00:38.860]  How many other people remember that one? There were 16
[00:39.140 --> 00:42.560]  speakers there, if I have to remember that.
[00:42.840 --> 00:46.820]  I have an emphasis in the blue team. However, I'm a formal purple teamer
[00:47.160 --> 00:50.880]  and all this stuff. And I'm in Pittsburgh now, but moving to
[00:50.880 --> 00:54.760]  Kirkland, Washington very, very soon. You will always
[00:54.760 --> 00:58.800]  find me sipping Grand Mayan Extra Añejo, or in
[00:58.800 --> 01:03.740]  this case, it is Casanoble Añejo.
[01:03.740 --> 01:06.840]  So, cheers. Let's kick off DEFCON. Jason
[01:06.840 --> 01:08.960]  already did a great job.
[01:11.120 --> 01:15.240]  And also, special thanks to the entire Red Team Village,
[01:15.240 --> 01:18.720]  all the staff, all the volunteers, everybody for kicking butt
[01:18.720 --> 01:22.980]  and making this happen. A lot of my favorite stuff are honey pots
[01:22.980 --> 01:26.640]  and refrigerators. On Saturday, I'm doing a talk for
[01:26.640 --> 01:30.920]  IoT security, so check that out, too. But this is one of my favorite ones
[01:30.920 --> 01:34.300]  because I just finished writing this particular
[01:35.560 --> 01:39.380]  presentation just about a month ago, made some changes to it, and so on.
[01:39.380 --> 01:43.000]  Remember, please, the views and opinions here are mine and mine
[01:43.000 --> 01:47.000]  alone. They do not reflect anything from my employers, past
[01:47.000 --> 01:50.940]  or present. Anything that I show you,
[01:50.940 --> 01:54.780]  as always, please use with a grain of salt. Don't go breaking
[01:54.780 --> 01:58.820]  into things that you shouldn't be breaking into. And those of you with an
[01:58.820 --> 02:02.320]  overwhelming fear of the unknown, you'll be happy to know that
[02:02.320 --> 02:06.720]  there is no hidden message in the disclaimer if you read it
[02:06.720 --> 02:10.340]  backwards. Yes, my jokes are that bad.
[02:10.380 --> 02:14.920]  Alright, let's get this thing on the road. So, why are we
[02:14.920 --> 02:18.760]  not here? Well, first of all, I'm not going to solve all
[02:18.760 --> 02:22.640]  your Kubernetes security woes. I wish I could.
[02:22.640 --> 02:27.180]  I wish Kubernetes and containers and everything were secure by default.
[02:27.180 --> 02:30.980]  They are not. Neither is the person
[02:30.980 --> 02:34.520]  sitting next to you, although, well, that doesn't work anymore because
[02:34.520 --> 02:38.520]  nobody's sitting... well, maybe somebody's sitting next to you. You never know.
[02:38.520 --> 02:42.880]  You never know. Okay. And common sense went out the window
[02:42.880 --> 02:46.540]  decades ago. Think about that. When it comes to
[02:46.540 --> 02:50.940]  information security, cyber security, hacking, whatever you want to call it,
[02:50.940 --> 02:54.340]  I honestly think we lost common sense quite
[02:54.340 --> 02:58.560]  some time ago. Why really are
[02:58.560 --> 03:02.520]  we here? Well, Kubernetes, I
[03:02.520 --> 03:06.740]  don't know. I still think it's fairly new. It's fun.
[03:06.740 --> 03:10.420]  It's crazy. It's a lot of things.
[03:10.420 --> 03:14.560]  You can deploy it in clouds. You can deploy it in...
[03:14.560 --> 03:18.500]  I have Raspberry Pis sitting down here. I was
[03:18.500 --> 03:22.280]  going to be showing that. There have been some demo problems and also some
[03:22.280 --> 03:26.520]  timing issues, so I won't be showing my Raspberry Pi cluster today.
[03:26.520 --> 03:30.160]  But you can build very extensive Raspberry Pi
[03:30.160 --> 03:34.780]  clusters and do all the things that I'm going to talk about today.
[03:35.340 --> 03:38.800]  Containers. Containers are not new.
[03:38.800 --> 03:42.400]  But people seem to have forgotten that they
[03:42.400 --> 03:46.460]  have anything to do with security. They get all
[03:46.460 --> 03:50.500]  hyped up on Kubernetes security, and they're forgetting about
[03:50.500 --> 03:54.240]  containers. We're going to talk about containers because it's
[03:54.240 --> 03:58.600]  important. It's, in fact, very important. And always remember this.
[03:58.600 --> 04:02.480]  Security is for everyone. Common
[04:02.480 --> 04:06.080]  sense is required. So please
[04:06.080 --> 04:10.340]  disregard the previous slide. Throw it out the window. Why?
[04:10.340 --> 04:14.560]  Because if we're not using common sense in security, then
[04:14.560 --> 04:18.220]  we're definitely not going to get where we need to be.
[04:18.640 --> 04:22.380]  All right. Just some quick security facts. We want to
[04:22.380 --> 04:26.640]  know this stuff. In 2018 alone, $114
[04:26.640 --> 04:30.620]  billion were spent by companies on
[04:30.620 --> 04:34.580]  information security. On fancy software,
[04:34.580 --> 04:39.100]  fancy hardware, things with blinky lights and all of that.
[04:39.100 --> 04:42.980]  But breaches kept on going up. We're doing
[04:42.980 --> 04:47.120]  something wrong here. Security appliances and software
[04:47.120 --> 04:51.060]  have vulnerabilities that we're not dealing with or that
[04:51.060 --> 04:55.320]  need to be dealt with by the vendors. Kubernetes itself
[04:55.320 --> 04:59.700]  has had a few issues. We'll talk about that. But so has Docker.
[04:59.700 --> 05:03.160]  So don't forget your containers. Also, I've
[05:03.160 --> 05:07.400]  actually had people say to me, we have a very unique security
[05:07.400 --> 05:11.460]  architecture. And I'm like, no, you don't.
[05:11.560 --> 05:15.280]  We all do the same stuff. So remember, your architecture
[05:15.280 --> 05:19.360]  is not unique. I love this quote. If you don't have this
[05:19.360 --> 05:23.500]  book, Offensive Countermeasures by John Strand, go out
[05:23.500 --> 05:27.220]  and get it. It's a small book with a ton of information.
[05:27.220 --> 05:31.280]  So get it. But he said, instead of brilliance,
[05:31.280 --> 05:35.840]  we've standardized on mediocrity. And that seems to be
[05:35.840 --> 05:40.100]  what information security is about these days.
[05:40.100 --> 05:43.800]  And I kind of have issues with that. So let's talk about
[05:43.800 --> 05:48.040]  that some more. First of all, let's get a common definition
[05:48.040 --> 05:51.920]  of what is a breach. Or maybe we'll call it
[05:51.920 --> 05:56.200]  what is an incident. Let's not necessarily call it a breach.
[05:56.200 --> 05:59.520]  So most incidents are not caused by zero-day
[05:59.520 --> 06:04.620]  fancy stuff that you find on the Internet. Typically,
[06:04.620 --> 06:08.160]  they're done because people have made mistakes and so on.
[06:08.160 --> 06:12.140]  Most breaches are not coming from your vulnerability scanners,
[06:12.140 --> 06:16.120]  especially when it comes to Kubernetes. You're not going to run a vulnerability
[06:16.120 --> 06:20.360]  scanner and go, oh, look, fix this one thing, and we're
[06:20.360 --> 06:24.120]  done and ready to go. No, it doesn't happen that way.
[06:25.420 --> 06:28.000]  Most breaches are going to come from
[06:28.580 --> 06:32.380]  configuration issues. This is something I want
[06:32.380 --> 06:36.160]  to emphasize. In fact, I'm going to show you some examples how
[06:36.160 --> 06:40.380]  configuration issues tend to cause most of the breaches. A close
[06:40.380 --> 06:44.460]  second is compromised credentials.
[06:44.500 --> 06:49.040]  When I say credentials, I still go out on GitHub.
[06:49.240 --> 06:52.280]  Well, maybe not every single day. And I think Jason
[06:52.280 --> 06:56.080]  even mentioned it, too, in his previous talk on bug bounties.
[06:56.320 --> 07:00.540]  Go out and look on GitHub and search for simple things
[07:00.540 --> 07:04.520]  like password. See how many times you find
[07:04.520 --> 07:08.520]  the word root. Yes, thousands of them
[07:08.520 --> 07:12.700]  are invalid. They don't make any sense. It's probably going to find the keyword
[07:12.700 --> 07:17.140]  root or password, and it's not going to get you anywhere.
[07:17.140 --> 07:20.540]  And yet I still find where people hard-code credentials
[07:20.540 --> 07:24.540]  into variables that say password. And right
[07:24.540 --> 07:28.600]  next to it, there's the key. There's the whatever the password
[07:28.600 --> 07:32.560]  is. Also, trailing in third
[07:32.560 --> 07:36.780]  place for most incidents is overprivileged
[07:36.780 --> 07:40.620]  users. This is incredibly important in
[07:40.620 --> 07:44.480]  the Kubernetes world. I'm going to show you this in just a second
[07:44.480 --> 07:49.160]  or in a couple of slides here. So keep these three things in mind.
[07:49.160 --> 07:52.920]  Configuration issues, compromised credentials, and overprivileged
[07:52.920 --> 07:56.800]  users. Okay. Kubernetes
[07:56.800 --> 08:00.820]  itself, we have to deal with all three
[08:00.820 --> 08:04.720]  of these compromises or these types
[08:04.720 --> 08:09.880]  of incidents that are going to cause issues in Kubernetes.
[08:09.960 --> 08:12.980]  But first, think
[08:12.980 --> 08:16.960]  about this. You know, we think about the Capital One hack
[08:16.960 --> 08:20.980]  that happened recently. Well, that was a combination
[08:20.980 --> 08:24.820]  of compromised credentials and access
[08:24.820 --> 08:29.000]  that should not have been granted. We see how
[08:29.000 --> 08:32.980]  Tesla, you know, they had poor access credentials
[08:32.980 --> 08:37.160]  and so on. Equifax, we all know about that one.
[08:37.160 --> 08:40.580]  Go back and check if you don't. You know, Equifax
[08:40.580 --> 08:44.300]  was hacked not only by a simple vulnerability
[08:44.820 --> 08:48.980]  that had not been patched, but also because they had very
[08:48.980 --> 08:53.040]  very poor lateral movement detection on the
[08:53.040 --> 08:57.000]  inside. Once they got in with more compromised
[08:57.000 --> 09:01.240]  credentials through the vulnerability, they weren't
[09:01.240 --> 09:05.180]  detected for quite some time. This is going to be
[09:05.180 --> 09:09.220]  critical. And one other thing, I was bored this morning. I decided
[09:09.220 --> 09:13.140]  to throw this up here in this slide. Why? Well, I
[09:13.140 --> 09:17.060]  put up a new spice rack in my kitchen. I really liked how it looks.
[09:17.060 --> 09:21.220]  I took a picture and I added it to the slide. There it is.
[09:21.220 --> 09:25.240]  Isn't it a great spice rack? Come on. Alright, you all better
[09:25.240 --> 09:29.460]  be laughing. Alright, let's talk about containers.
[09:30.180 --> 09:33.160]  Quick review of containers. They are not
[09:33.160 --> 09:37.400]  secure by default. Now, I'm really focusing more on
[09:37.400 --> 09:40.980]  Docker containers for now because that tends to be
[09:40.980 --> 09:45.040]  the norm. And one thing to think about when it comes to
[09:45.040 --> 09:48.760]  containers, containers are not like
[09:50.000 --> 09:52.740]  a VM. A VM is actually more
[09:52.740 --> 09:56.840]  isolated. And this is something that's going to be important coming
[09:56.840 --> 10:00.760]  up here. So, a container is a grouping of stuff
[10:00.760 --> 10:04.660]  and the stuff that we like to call it are resources. Well, the
[10:04.660 --> 10:08.820]  resources are grouped into namespaces. If
[10:08.820 --> 10:12.740]  you are new to containers, remember this. Namespaces
[10:12.740 --> 10:16.860]  are critical. You group your processes,
[10:16.860 --> 10:20.980]  your networks, your users, your ports, your interprocess. All
[10:20.980 --> 10:24.680]  of this stuff can be controlled and
[10:24.680 --> 10:28.960]  locked down within namespaces. Resources
[10:28.960 --> 10:32.840]  are exploitable. I was sitting in another
[10:33.980 --> 10:36.640]  web broadcast just the other day
[10:36.640 --> 10:40.240]  on containers and I was
[10:41.660 --> 10:44.740]  surprised because it was supposed to be a more
[10:44.740 --> 10:48.820]  intermediate presentation on containers and
[10:48.820 --> 10:53.060]  yet all of the people were asking is there a
[10:53.060 --> 10:56.740]  way to prevent process
[10:57.560 --> 11:00.960]  abuse such as CPUs and memory and
[11:00.960 --> 11:04.740]  so on. And again, this was supposed to be an intermediate
[11:05.560 --> 11:09.100]  class on containers so it
[11:09.100 --> 11:12.860]  confused me a little bit that people were asking that because that should be something
[11:12.860 --> 11:17.740]  that you know from the very beginning that you get into containers.
[11:17.740 --> 11:20.920]  Using namespaces and the next
[11:20.920 --> 11:25.120]  most important thing is control groups or cgroups. These
[11:25.120 --> 11:29.460]  are the limits that you are going to be putting on containers.
[11:29.460 --> 11:32.820]  If you don't put limits on them, you're
[11:32.820 --> 11:37.180]  going to run into problems. Every time I do a pen test
[11:37.180 --> 11:41.320]  of a new application deployed into a container,
[11:41.320 --> 11:45.500]  I would say that 7 out of 10 times
[11:45.500 --> 11:49.340]  I find that the author of the application
[11:49.340 --> 11:53.220]  who has put it in a container has not put any limits.
[11:53.220 --> 11:57.540]  I'll get into the application in whatever way possible whether it's
[11:58.140 --> 12:01.180]  a SQL injection or an RCE
[12:01.180 --> 12:05.400]  or something like that. And once I'm in, I can take over not just the
[12:05.400 --> 12:09.040]  container but the entire system that it's on
[12:09.040 --> 12:13.800]  because there are no cgroups applied to the container.
[12:13.800 --> 12:17.120]  The other thing to remember about containers
[12:17.120 --> 12:20.960]  is think of a container as an image or more
[12:20.960 --> 12:25.880]  correctly, think of it as a file system snapshot.
[12:26.060 --> 12:28.880]  So who decides what goes
[12:28.880 --> 12:34.220]  in that image? If you look at this particular picture,
[12:34.220 --> 12:36.900]  you'll see, you know, here we have
[12:36.900 --> 12:40.200]  namespaces, the control groups that I show.
[12:40.480 --> 12:44.780]  It's going to possibly be cherooted if you've set it up
[12:44.780 --> 12:48.860]  correctly. And here's the application.
[12:49.400 --> 12:52.840]  There's a Docker client that you might use to launch
[12:52.840 --> 12:57.120]  this. There's a Docker registry. This is critical
[12:57.120 --> 13:00.940]  to the security of your system. If you're
[13:00.940 --> 13:05.320]  new to Docker, you may not know this, but Docker Hub was compromised
[13:05.320 --> 13:09.400]  two years ago, I believe it was. I've lost track
[13:09.400 --> 13:12.520]  of time during COVID. I don't know anymore.
[13:12.860 --> 13:17.160]  But you want to have a secure registry where you keep
[13:17.160 --> 13:21.000]  your images because if they're somewhere
[13:21.000 --> 13:25.800]  that you don't have control, you never know what's going to happen.
[13:25.800 --> 13:29.020]  But here's the difference. Notice that we have a host OS
[13:29.020 --> 13:34.280]  and hardware down here, unlike a VM.
[13:34.280 --> 13:38.600]  In a VM, you would see the entire operating system
[13:38.600 --> 13:42.520]  within the VM here. You would not see a shared
[13:42.520 --> 13:46.560]  OS. Yes, there's a host OS, but then you also have the guest
[13:46.560 --> 13:50.580]  operating systems themselves. And that's where we get into
[13:50.580 --> 13:54.580]  how a VM is much more isolated
[13:54.580 --> 13:58.300]  and contained than a container is. So
[13:58.300 --> 14:02.540]  don't think the word container means that you're more secure.
[14:02.540 --> 14:06.460]  It just means that we have some work to do. So that's why
[14:06.460 --> 14:10.120]  I say right here in this slide, container isolation.
[14:10.320 --> 14:14.740]  Is it really isolated? The simple answer to that is
[14:14.740 --> 14:18.500]  no, it is not. So let's talk about some
[14:18.500 --> 14:22.700]  container best practices. I should have also said at the very
[14:22.700 --> 14:26.540]  beginning of this talk, this is an intro.
[14:26.540 --> 14:30.360]  I can't show you everything. I'm trying to get people to start
[14:30.360 --> 14:34.320]  thinking about how you secure containers, then
[14:34.320 --> 14:38.340]  we're going to jump into how you secure Kubernetes at the
[14:38.340 --> 14:42.320]  same time. So remember, we have
[14:42.520 --> 14:46.300]  a base image. That means that base image has to come from
[14:46.480 --> 14:50.000]  a secure repo. If you do not know what CIS
[14:50.000 --> 14:54.200]  is, definitely go to CIS
[14:54.200 --> 14:57.820]  or CISecurity.org
[14:58.460 --> 15:01.860]  Check it out because you want to use CIS
[15:01.860 --> 15:05.900]  benchmarks for locking down your base image. I'm going to
[15:05.900 --> 15:10.020]  talk about that a little more coming up. Don't run your images
[15:10.020 --> 15:14.040]  as root. By default, if you do not set a user
[15:14.040 --> 15:17.920]  name, your container and your application in the container is going
[15:17.920 --> 15:21.480]  to run it as root. I have found so many
[15:21.480 --> 15:26.040]  who have deployed WordPress applications
[15:26.040 --> 15:29.220]  in a Docker container via Kubernetes
[15:29.980 --> 15:34.000]  and they don't set the user name and WordPress is running
[15:34.000 --> 15:37.960]  as root, so is the database and everything else. So of course,
[15:37.960 --> 15:42.400]  once it's compromised, you take over everything. Also,
[15:42.400 --> 15:46.160]  visibility. You need monitoring within
[15:46.160 --> 15:49.880]  your containers. Your containers put a lot of information
[15:49.880 --> 15:54.180]  out and they can log it, but you have to capture that. We're going to talk
[15:54.180 --> 15:58.220]  about that in a minute. Also, think of
[15:58.220 --> 16:02.140]  it as a domino. If one container is
[16:02.140 --> 16:06.280]  compromised, I guarantee you the rest of them will
[16:06.280 --> 16:09.260]  fall over like dominoes.
[16:09.880 --> 16:14.520]  When you start to get into containers and Kubernetes, you're going to see this.
[16:14.920 --> 16:18.860]  Also, we want to set some controls.
[16:18.900 --> 16:22.040]  Besides setting a user name that is not
[16:22.040 --> 16:25.940]  root, we also set privileges and we set
[16:25.940 --> 16:29.940]  certain attributes to those users where it says
[16:29.940 --> 16:33.780]  no elevated privileges or no new privileges.
[16:33.780 --> 16:37.720]  You don't want user accounts or IDs
[16:37.720 --> 16:42.020]  to be able to escalate privileges. And I'll
[16:42.020 --> 16:46.080]  show you some examples coming up. This one, my god, this is
[16:46.080 --> 16:50.020]  near and dear to my heart, secrets management. I still
[16:50.020 --> 16:54.180]  find, I would love to say by a show of hands,
[16:54.180 --> 16:58.040]  but I can't do that right here, how many people
[16:59.480 --> 17:02.060]  put the secrets
[17:02.060 --> 17:05.760]  and tokens and things like that in their Docker
[17:05.760 --> 17:09.720]  YAML files so when they then deploy the Docker
[17:09.720 --> 17:14.040]  image, if somebody gets a hold of their YAML file, which
[17:14.040 --> 17:18.340]  describes the configuration of that container,
[17:18.340 --> 17:22.340]  well, the secrets are right in it. I've worked
[17:22.340 --> 17:26.460]  with a lot of developers and a lot of engineers and I get
[17:26.460 --> 17:30.320]  them, you know, you have to start thinking about putting
[17:30.320 --> 17:34.360]  your secrets in another place, whether it's HashiCorp
[17:34.360 --> 17:38.240]  Vault, whether it's a cloud secrets management system,
[17:38.240 --> 17:42.420]  something. But get them out of the apps and get them out of the
[17:42.420 --> 17:46.480]  configuration files. And stop allowing SSH
[17:46.480 --> 17:49.740]  into your hosts. When we get into
[17:50.660 --> 17:54.340]  Kubernetes, coming up in a couple slides here,
[17:54.340 --> 17:58.600]  one of the key constructs of Kubernetes is you have a master node
[17:58.600 --> 18:02.660]  or master nodes, and then you have worker nodes. And I
[18:02.660 --> 18:06.500]  still find worker nodes with SSH enabled
[18:06.500 --> 18:10.380]  on them so people can SSH in and, quote, fix
[18:10.380 --> 18:14.640]  things. No, no, you use the
[18:14.640 --> 18:18.820]  management tools to fix things and redeploy. Stop
[18:18.820 --> 18:22.940]  allowing SSH into your systems. Also, I can't
[18:22.940 --> 18:26.880]  stress it enough times, namespace
[18:26.880 --> 18:30.840]  limits. Go read up on how to
[18:30.840 --> 18:34.820]  control namespaces. That is going to
[18:34.820 --> 18:38.800]  be one of your best efforts and best
[18:38.800 --> 18:42.760]  practices for containing and controlling and
[18:42.760 --> 18:46.320]  securing a container before you distribute it to
[18:47.240 --> 18:50.780]  or deploy it via Kubernetes. This is one of
[18:50.780 --> 18:54.800]  my favorite tools. It's called Docker Slim. If you've never heard
[18:54.800 --> 18:58.760]  of it, if I were to create a container,
[18:58.760 --> 19:02.340]  let's say I create a container, it has Ubuntu in it
[19:02.340 --> 19:06.260]  or the base of Ubuntu, and then maybe it has WordPress
[19:06.900 --> 19:11.020]  running in it for whatever god awful reason that I want to do this.
[19:11.020 --> 19:15.180]  Let's say that entire image ends up being
[19:15.180 --> 19:18.700]  three to four hundred meg. It's still
[19:18.700 --> 19:23.300]  manageable, but if I were to run that through Docker Slim,
[19:23.300 --> 19:27.400]  not only will it add some extra configurations
[19:27.400 --> 19:31.380]  and namespace limits and so on, but it shrinks it down
[19:31.380 --> 19:34.840]  and something that might be three or four hundred megabytes
[19:34.840 --> 19:39.180]  deployed into a container will get shrunk down to
[19:39.180 --> 19:43.200]  maybe 10, 15, 20 meg. This
[19:43.200 --> 19:46.960]  makes it even that much more manageable. It's easier to
[19:46.960 --> 19:50.880]  deploy. It's quicker to deploy, especially
[19:50.880 --> 19:55.260]  if there are requirements for scaling.
[19:55.260 --> 19:58.680]  So think about this. Take a look at this. I also mentioned
[19:58.680 --> 20:03.000]  in Discord that my slides will be available in my GitHub
[20:03.000 --> 20:07.140]  repo, and I'll get them to the Red Team Village so they can add them
[20:07.140 --> 20:12.000]  to the Google Docs location after my talk.
[20:13.260 --> 20:15.340]  And always remember
[20:15.340 --> 20:19.200]  apply SELinux or AppArmor. One of the things
[20:19.200 --> 20:23.120]  that Docker Slim does is it will do that for you.
[20:23.120 --> 20:27.080]  It's very easy to enable SELinux or AppArmor
[20:27.080 --> 20:31.300]  within a container. If you can do that, you're going to make
[20:31.300 --> 20:35.220]  it that much more secure. It's not on by default, but you can set
[20:35.220 --> 20:36.960]  it up so it will be.
[20:39.140 --> 20:44.040]  And remember the infrastructure, as I put it.
[20:44.640 --> 20:45.400]  You
[20:47.120 --> 20:51.180]  can't forget that all of this stuff is being deployed on
[20:51.180 --> 20:55.160]  an infrastructure of some sort, and that infrastructure better
[20:55.160 --> 20:59.180]  be secure. I mentioned SSH. People keep
[20:59.180 --> 21:03.440]  allowing SSH into their environment. Be very
[21:03.440 --> 21:07.500]  aware that your infrastructure itself should be secure. And I will
[21:07.500 --> 21:11.280]  remind you later about that. All right. Here's a quick picture.
[21:11.280 --> 21:15.720]  Picture is worth a thousand words, right? If you look at our first example,
[21:15.720 --> 21:19.360]  this is our monolithic operating system here. We see
[21:19.360 --> 21:23.560]  hardware and operating system. We run apps. Here we see a VM. So we see
[21:23.560 --> 21:27.300]  hardware operating system. There's a hypervisor involved.
[21:27.900 --> 21:31.320]  And then there's going to be virtual machines. But when you have virtual
[21:31.320 --> 21:35.100]  machines, you have additional operating systems, you have additional
[21:35.100 --> 21:39.480]  libraries. They're very much isolated. I should have used a different
[21:39.480 --> 21:43.260]  picture where there's a big line through it that
[21:43.260 --> 21:47.040]  shows, yes, these are isolated and so on. But
[21:47.040 --> 21:51.380]  that should not have happened. Nope. Go back.
[21:51.540 --> 21:54.720]  Sorry. Okay. But when we look at
[21:55.320 --> 21:59.300]  using Docker or Container, we go hardware operating
[21:59.300 --> 22:03.680]  system. There is a container runtime. There's no real dividing lines.
[22:03.680 --> 22:07.400]  And then we see containers. The only thing that is unique between
[22:07.400 --> 22:11.380]  the containers is that bin library, the things
[22:11.380 --> 22:15.340]  that are going to run in the application. So there's not a lot of
[22:15.340 --> 22:19.360]  isolation there. And people want to
[22:19.360 --> 22:23.460]  think that there is. But there is not. So please keep
[22:23.460 --> 22:27.080]  that in mind. I do love this
[22:27.080 --> 22:30.940]  comic. It came out a couple of years ago. A great
[22:30.940 --> 22:35.040]  defense is, you know, we'll inundate them
[22:35.040 --> 22:39.100]  with quarterly Kubernetes releases. Yes, they do update
[22:39.100 --> 22:42.820]  Kubernetes a lot. That's why you have to pay
[22:42.820 --> 22:46.680]  attention to your environment and the tools that you're going to use
[22:46.680 --> 22:51.140]  so you can do upgrades to your Kubernetes cluster. So let's bring on
[22:51.140 --> 22:55.020]  the orchestrator. Let's talk about Kubernetes. All right. Here
[22:55.020 --> 22:58.840]  is Kubernetes 101. In one slide, I'm going to teach you
[22:58.840 --> 23:02.780]  at least the basics of how Kubernetes
[23:02.780 --> 23:07.020]  works. We have a master node. The master node is
[23:07.020 --> 23:10.560]  critical. It manages all of the Kubernetes
[23:10.560 --> 23:14.520]  worker nodes. It manages the clusters. It deploys
[23:14.520 --> 23:18.460]  pods. Think of a pod as a collection
[23:19.020 --> 23:23.000]  of containers that are grouped.
[23:23.000 --> 23:26.960]  A pod might be the way that you
[23:26.960 --> 23:30.880]  decide, oh, I want to have a pod that supports a particular
[23:30.880 --> 23:34.880]  client. Or maybe it's a WordPress instance. And that
[23:34.880 --> 23:39.280]  pod has WordPress. It has a container, a
[23:39.280 --> 23:43.140]  separate container for the database and so on. So
[23:43.140 --> 23:47.320]  the pod is how you can group things that
[23:47.320 --> 23:51.180]  are various containers into certain groupings.
[23:51.180 --> 23:55.260]  And those groupings are kind of defined by you. But stop and think about
[23:55.260 --> 23:58.920]  how you want to deploy pods. It's going to be important.
[23:58.920 --> 24:03.580]  A worker node. The node is where all the containers are going to run.
[24:03.580 --> 24:07.200]  You have other Kubernetes components that have to
[24:07.200 --> 24:11.020]  run there. But the important part is, this is where the containers
[24:11.020 --> 24:14.760]  run. If the worker node is not secure
[24:14.760 --> 24:19.020]  and not patched, and I can get into that
[24:19.020 --> 24:22.640]  underneath the visibility of
[24:22.640 --> 24:27.000]  Kubernetes, then Kubernetes has no idea that I've compromised
[24:27.500 --> 24:31.160]  a node. I found numerous Kubernetes sites around
[24:31.160 --> 24:34.540]  the Internet where there's
[24:34.540 --> 24:39.300]  additional applications running on worker nodes because they've been compromised.
[24:39.300 --> 24:43.080]  Of course, they're Bitcoin miners. But the master node has no
[24:43.080 --> 24:46.980]  idea that it's running on there because they got in via
[24:46.980 --> 24:50.740]  SSH and they launched the application, the
[24:50.740 --> 24:54.900]  miner, underneath the visibility of Kubernetes.
[24:54.900 --> 24:59.100]  So you really want to lock down these worker nodes. A pod,
[24:59.100 --> 25:02.900]  I already explained, it's a unit of deployment and it's also how
[25:02.900 --> 25:07.080]  you address things from Kubernetes. And a pod
[25:07.080 --> 25:10.520]  will have an IP address. So that's important to
[25:10.520 --> 25:15.080]  remember. But it's going to contain one or more containers.
[25:15.080 --> 25:18.980]  I like to think of pods as, even though it might have
[25:18.980 --> 25:22.840]  its own IP address, but think of it like a VLAN in a way.
[25:22.840 --> 25:26.900]  Because I've isolated it in one way. Services
[25:26.900 --> 25:30.200]  are various things that run within Kubernetes
[25:30.980 --> 25:34.920]  that communicate, that balance, that do a lot of work. We're going to look
[25:34.920 --> 25:38.840]  at services coming up. There are various components that are
[25:38.840 --> 25:42.760]  used. Three that are critical, the API
[25:42.760 --> 25:46.640]  server, you're going to have kubelets, and you're also
[25:46.640 --> 25:51.000]  etcd. The one concept you should take
[25:51.000 --> 25:55.040]  about etcd is this is the database
[25:55.040 --> 25:58.700]  that describes and builds everything
[25:58.700 --> 26:02.580]  that is your Kubernetes cluster. The etcd
[26:02.580 --> 26:06.280]  environment should actually be walled off
[26:06.280 --> 26:10.560]  and firewalled and controlled very, very
[26:10.560 --> 26:14.580]  securely. If you do that, you will have a much
[26:14.580 --> 26:18.720]  more secure Kubernetes cluster. So keep
[26:18.720 --> 26:23.040]  that in mind. And you can see it here by looking at a picture.
[26:23.040 --> 26:26.420]  So we take all of those concepts that we just talked about
[26:26.420 --> 26:31.000]  we see here's an ABI server, here's a controller,
[26:31.000 --> 26:33.660]  this is all part, you know, the kubectl...
[26:33.660 --> 26:38.640]  Stop doing that. Come on. Kubectl
[26:38.640 --> 26:42.640]  is the command you use to control everything.
[26:43.020 --> 26:46.580]  And even though this is a disk kind of a representation
[26:46.580 --> 26:50.740]  of etcd, I like to say, you know what, this needs to be off
[26:50.740 --> 26:54.920]  on its own, firewalled, and separated, and it can be done that way.
[26:54.920 --> 26:58.420]  Here's your worker node, and you can see how you might have multiple
[26:58.420 --> 27:02.440]  pods on a worker node. This pod might be talking to
[27:02.440 --> 27:06.620]  this pod. It doesn't necessarily have to talk to that one, or
[27:06.620 --> 27:10.720]  they may not talk at all. These are some of the other services that are running
[27:10.720 --> 27:14.540]  the proxies. This is where we are exposed to the internet,
[27:14.540 --> 27:18.500]  not over here. Okay? So the only place you're
[27:18.500 --> 27:22.240]  exposing Kubernetes is at the worker nodes.
[27:22.780 --> 27:26.280]  Now, Kubernetes is not perfect.
[27:26.520 --> 27:29.980]  There have been problems. There was...
[27:29.980 --> 27:34.640]  I need to update this slide again, because there have been a couple of other
[27:34.640 --> 27:38.900]  Kubernetes. These are more for reference. One's a
[27:38.900 --> 27:42.800]  little older. This was back... there was the backdoor that involved the
[27:42.800 --> 27:46.760]  API server. There were some others. These are only for your reference,
[27:46.760 --> 27:51.080]  so you can go look them up and see. And there are
[27:51.080 --> 27:54.720]  more. The whole point I'm trying to make here is
[27:55.300 --> 27:59.480]  Kubernetes is not secure by default, and it's not
[27:59.480 --> 28:02.760]  perfect. Well, let's try to make
[28:02.760 --> 28:06.500]  it perfect. Let's take a common Kubernetes cluster
[28:06.500 --> 28:10.920]  and say, you know, if we were to threat model this,
[28:10.920 --> 28:14.680]  and that is very, very important, first thing
[28:14.680 --> 28:18.800]  we want to do is to say, TLS everywhere. Yeah, I know, I have
[28:18.800 --> 28:22.620]  too many exclamation points, but I'm trying to make the point. If
[28:23.640 --> 28:26.820]  everything within your Kubernetes cluster is using TLS
[28:26.820 --> 28:30.640]  and is encrypted, then it's that much
[28:30.640 --> 28:34.700]  safer. I still have engineers come to me and say,
[28:34.700 --> 28:38.620]  our application is completely internal. Why do we
[28:38.620 --> 28:42.680]  need TLS? It's never going to be exposed to the internet.
[28:42.680 --> 28:46.560]  I said, really? So what happens if Joe's application
[28:46.560 --> 28:50.680]  over here, which is exposed to the internet, is compromised, and then
[28:50.680 --> 28:55.000]  they use that as a pivot point to gain access to this environment,
[28:55.000 --> 28:58.380]  and then they compromise your system? O
[28:58.380 --> 29:02.480]  is typically the response. So I can't say it
[29:02.480 --> 29:06.440]  enough, harden the bloody infrastructure. Yes, I'm saying
[29:06.440 --> 29:11.060]  it again, but you're running this thing on your infrastructure,
[29:11.060 --> 29:14.320]  make sure it's hardened. And very important,
[29:14.320 --> 29:18.180]  two concepts. There's RBAC, which is
[29:18.180 --> 29:22.780]  regular role-based access control. Set that up.
[29:22.780 --> 29:26.000]  Make sure you're doing it right. Disable
[29:26.000 --> 29:28.600]  ABAC. ABAC is
[29:30.300 --> 29:33.960]  incredibly difficult to manage. You
[29:33.960 --> 29:38.000]  will spend days and hours and hours trying
[29:38.000 --> 29:41.920]  to do ABAC access controls,
[29:41.920 --> 29:46.000]  which is more granular, but it's not worth the effort.
[29:46.000 --> 29:50.060]  Make sure you use RBAC and manage it correctly.
[29:50.060 --> 29:53.700]  And I'm going to show you a couple of tools coming up
[29:53.700 --> 29:58.000]  on how to check it. Also, check out this blog
[29:58.000 --> 30:01.920]  entry. I mentioned this previously about
[30:01.920 --> 30:05.520]  monitoring. This is one way to audit
[30:05.520 --> 30:09.560]  Kubernetes, so keep this in mind. Take a look at that
[30:09.560 --> 30:13.140]  blog entry from Wazoo, and it's an excellent
[30:14.780 --> 30:17.680]  place to start when it comes to monitoring
[30:17.680 --> 30:21.960]  your environment. Also, kubectl, get role
[30:21.960 --> 30:25.960]  binding in all your namespaces. You should do this
[30:25.960 --> 30:29.280]  periodically in your environment, especially
[30:29.720 --> 30:33.680]  if you have engineers deploying their own
[30:33.680 --> 30:37.920]  applications. Make sure that all the namespaces
[30:37.920 --> 30:42.300]  have good controls in their RBAC accounting.
[30:42.300 --> 30:46.040]  Also, use third-party authentication for the API
[30:46.040 --> 30:50.000]  server. Again, something like HashiCorp Vault. Yes, I love
[30:50.000 --> 30:54.100]  that tool. If you are not familiar with it, get familiar with
[30:54.100 --> 30:58.000]  it. It's a great tool, and it integrates very, very well with your
[30:58.000 --> 31:00.360]  API server and Kubernetes.
[31:01.680 --> 31:06.120]  I already mentioned this. Separate and firewall your etc cluster. Make sure
[31:06.120 --> 31:09.800]  it's protected. That is the keys to the kingdom.
[31:09.880 --> 31:13.700]  It's the entire configuration of your cluster. And rotate
[31:13.700 --> 31:17.880]  encryption keys. Yes, use Vault. And by
[31:18.380 --> 31:22.860]  default, turn on security policies in the pods.
[31:22.880 --> 31:25.860]  It's some simple lines of configuration
[31:25.860 --> 31:29.440]  in the YAML files. It's not hard to do. Or
[31:29.440 --> 31:33.920]  use Docker Slim, that tool that I mentioned earlier, because
[31:33.920 --> 31:38.000]  that will help you set this up. Alright, let's talk
[31:38.000 --> 31:41.920]  about risk. kubectl, get secrets. That
[31:41.920 --> 31:45.860]  is the one that I just mentioned where you can look at the secrets
[31:45.860 --> 31:49.840]  from pretty much everything. But here's some tools
[31:49.840 --> 31:53.740]  that I absolutely love by CyberArk. One is called Kubernetes
[31:53.740 --> 31:58.260]  RBAC audit. The other is called KubeScan.
[31:58.260 --> 32:02.100]  The RBAC audit does exactly what you think it does. It audits
[32:02.100 --> 32:05.560]  all of your RBAC roles. The KubeScan
[32:05.560 --> 32:10.000]  will look at all the permissions that are granted from
[32:10.000 --> 32:14.020]  the roles. These are two great tools that
[32:14.020 --> 32:17.860]  you want to put into your toolbox for securing
[32:17.860 --> 32:22.320]  Kubernetes clusters. Because remember at the very beginning, I said
[32:22.320 --> 32:26.320]  it's all about configurations. It's the number one
[32:26.320 --> 32:30.040]  area that Kubernetes clusters get
[32:30.040 --> 32:34.260]  compromised. Also, we want to take a look
[32:34.260 --> 32:38.260]  at GitHub. Go out and make sure
[32:38.260 --> 32:42.040]  that... take that away... go out and make sure that
[32:42.040 --> 32:46.260]  your own configurations are not being
[32:46.260 --> 32:50.880]  shared up to GitHub by your engineers accidentally.
[32:51.080 --> 32:54.340]  How many times does this happen? It happens.
[32:54.340 --> 32:58.580]  I've had it happen by engineers. They don't do it on purpose.
[32:58.580 --> 33:01.700]  But they upload code. And in the code are secrets.
[33:01.920 --> 33:05.540]  Oops. Oops is not a good thing.
[33:05.760 --> 33:10.400]  Also, you need a locked down secure image repo.
[33:10.400 --> 33:14.380]  And create gold images. Don't just let your
[33:14.380 --> 33:18.880]  engineers go out and do, hey, docker run
[33:18.880 --> 33:22.680]  bring down Ubuntu and pull it from the internet. No.
[33:22.680 --> 33:26.380]  You need your own repo and registry for your docker
[33:26.380 --> 33:30.280]  images. Make sure that happens. And please secure the
[33:30.280 --> 33:34.720]  libraries. That's another big one. We also...
[33:34.720 --> 33:39.120]  I need an entire slide on just the applications.
[33:39.940 --> 33:43.840]  I'm going out on a limb. This is my personal opinion.
[33:43.840 --> 33:47.180]  But I want to say that nine times out of ten... okay,
[33:47.180 --> 33:50.740]  maybe eight and a half. Eight and a half times out of ten
[33:50.740 --> 33:54.600]  Kubernetes clusters and containers are compromised
[33:55.120 --> 33:58.180]  because the applications themselves
[33:58.980 --> 34:02.860]  were poorly written and not tested.
[34:02.980 --> 34:06.900]  And that's what gets me. I have
[34:06.900 --> 34:10.720]  environments where I see the engineers create
[34:10.720 --> 34:14.660]  an application. They do so much work to lock down
[34:14.660 --> 34:18.880]  Kubernetes in the container, but they do a poor
[34:18.880 --> 34:23.260]  job and there's a SQL injection vulnerability in the application directly.
[34:23.360 --> 34:26.240]  So then the application is compromised
[34:26.900 --> 34:30.940]  and then the container is compromised because, oops,
[34:30.940 --> 34:35.500]  it was running as root and so on and so on. Remember the dominoes.
[34:35.500 --> 34:39.560]  Everything falls over. Also, remember your
[34:39.560 --> 34:43.420]  pods. Another common
[34:43.420 --> 34:47.160]  mistake that people make when they're new to Kubernetes
[34:47.160 --> 34:51.600]  is they don't stop and think, how can I
[34:51.600 --> 34:56.180]  group things into pods and secure those pods?
[34:56.180 --> 34:59.160]  At least if a container falls
[34:59.160 --> 35:03.380]  within a pod, then maybe there's a couple more containers
[35:03.380 --> 35:07.340]  in there. Those containers that get compromised are going to be somewhat
[35:07.340 --> 35:11.720]  limited within the pod itself. Typically
[35:12.060 --> 35:15.440]  a pod is not going to fall over when another pod falls
[35:15.440 --> 35:19.400]  over. It's usually containers within the pods. So you want to think
[35:19.400 --> 35:23.440]  about grouping things. Also, Jason mentioned
[35:23.440 --> 35:27.460]  this in his talk earlier, but showdown. If you have
[35:27.460 --> 35:31.300]  never used showdown, get familiar with it. Even
[35:31.300 --> 35:35.400]  for Kubernetes. You can go out to showdown. You can start
[35:35.400 --> 35:39.480]  searching for Kubernetes. You can search for API clusters
[35:39.480 --> 35:42.640]  or API servers. You can search
[35:43.140 --> 35:47.540]  for etcd servers. And if you find one of yours,
[35:47.840 --> 35:51.080]  oops, but they're out there.
[35:51.080 --> 35:54.800]  I did a search this morning for some etcd
[35:56.020 --> 35:57.540]  exposure and
[35:59.520 --> 36:01.660]  there were over 6,000
[36:01.660 --> 36:05.580]  that I found that were exposed. I don't know how many of those have
[36:05.580 --> 36:09.380]  vulnerabilities that would have allowed me to compromise them
[36:09.380 --> 36:13.440]  because I was just curious what the number would be.
[36:13.560 --> 36:17.600]  It goes up and down on a daily basis, but you should be
[36:17.600 --> 36:21.460]  aware of that. So use showdown. Let's talk about
[36:21.460 --> 36:25.860]  the configurations. I've said this before.
[36:25.860 --> 36:29.380]  Most of the breaches come from configuration issues.
[36:32.890 --> 36:35.950]  Use CIS hardening. Center for
[36:35.950 --> 36:39.950]  Internet Security. It is an amazing place. It is done by
[36:39.950 --> 36:43.990]  volunteers and workers within CIS. You can get
[36:43.990 --> 36:48.670]  benchmarks for Docker and you can get benchmarks for Kubernetes.
[36:48.670 --> 36:51.790]  You can use these to configure your environment
[36:51.790 --> 36:54.750]  so they are indeed hardened.
[36:55.710 --> 36:58.630]  Stop ignoring them. I still see people
[36:58.630 --> 37:02.390]  that tell me they've never heard of CIS and
[37:02.390 --> 37:06.850]  it makes me very sad when I hear that because it is such a great
[37:06.850 --> 37:10.930]  resource. The whole point here is
[37:10.930 --> 37:14.710]  we have to harden the environment from the
[37:14.710 --> 37:18.750]  standpoint of the basic configuration, but don't forget
[37:18.750 --> 37:22.930]  the application. Also, I know a lot of people
[37:22.930 --> 37:27.010]  have environments that are CICD when it
[37:27.010 --> 37:30.950]  comes to Docker and Kubernetes. That's why we have them. It's for
[37:30.950 --> 37:34.870]  speed. It's for development. It's for putting things out
[37:34.870 --> 37:38.930]  quickly. Okay, great. A lot of people use Jenkins.
[37:39.690 --> 37:42.550]  Is there some way to automate
[37:42.550 --> 37:47.150]  all of this so at least I can look at my images,
[37:47.150 --> 37:51.310]  I can look at my containers, all within the CICD pipeline?
[37:51.310 --> 37:55.350]  Yes, there is. There is a great tool called
[37:55.350 --> 37:59.570]  Anchor. Anchor, as it says here, is a
[38:00.030 --> 38:03.570]  container image scanner plugin. And what it does is
[38:03.910 --> 38:07.710]  you put it into the pipeline. It's a plugin for Jenkins.
[38:07.710 --> 38:11.770]  And it actually does security scans against the images
[38:11.770 --> 38:15.610]  that you're trying to deploy. This is a great way to
[38:15.610 --> 38:19.370]  get testing in before you actually
[38:19.370 --> 38:23.350]  deploy it to production. So let's talk about some
[38:23.350 --> 38:27.450]  additional tools before we get into the pen testing. Because we're still trying to
[38:27.450 --> 38:31.250]  lock it down. KubeBench. We want to run KubeBench.
[38:31.250 --> 38:35.910]  What it does is it runs against your CIS benchmarks.
[38:35.910 --> 38:36.790]  It displays
[38:39.870 --> 38:43.650]  a configuration that shows you what has passed, what has failed. I'll show an example
[38:43.650 --> 38:47.570]  in a minute. There's also KubeAudit. It's a little more granular
[38:47.570 --> 38:51.810]  than KubeSec. And if you run KubeBench
[38:51.810 --> 38:55.770]  and KubeAudit, it will give you a report that shows you
[38:55.770 --> 38:59.210]  oops, my system is insecure. Here's an example
[38:59.210 --> 39:03.370]  of a report. And when we look at it, we see allow
[39:03.370 --> 39:07.730]  privilege escalation is not set. There's a very
[39:07.730 --> 39:11.870]  first one. Oops, that needs to be turned on.
[39:11.870 --> 39:15.790]  Remember, that was the one that I mentioned that allows users
[39:15.790 --> 39:20.030]  to do privilege escalation. We don't want that. Oh, crap.
[39:20.030 --> 39:23.930]  We have run as non-root is not set. We don't have a read
[39:23.930 --> 39:27.590]  only file system set. There are so many
[39:27.590 --> 39:31.830]  simple things that can be done. Again, no app
[39:31.830 --> 39:35.830]  armor, no security comp. It's not being turned on.
[39:35.830 --> 39:39.750]  These are things that are just so easy. And this is from every
[39:39.750 --> 39:43.810]  single container within the environment that is
[39:43.810 --> 39:47.690]  being deployed under this particular. It's much
[39:47.690 --> 39:51.630]  longer. It goes into it. This is just a quick snapshot.
[39:51.630 --> 39:55.930]  The whole point here is you run this against your environment
[39:55.930 --> 39:59.710]  before you deploy it. And you go, oops, let's go back to our
[39:59.710 --> 40:02.970]  configuration file and fix all these things.
[40:03.670 --> 40:08.010]  Here are some more tools for you in the environment. Claire,
[40:08.010 --> 40:11.490]  another great tool for doing static analysis of
[40:11.490 --> 40:15.470]  applications. Claire does, it integrates
[40:15.470 --> 40:19.490]  Claire and your Docker registry so it can automate that
[40:19.650 --> 40:23.650]  a little bit. Falco, I need to put huge stars next
[40:23.650 --> 40:27.490]  to this one because this is actually one of my favorite, favorite
[40:27.490 --> 40:31.490]  tools. It was developed years
[40:31.490 --> 40:35.810]  ago and then it kind of morphed off and became falco.org.
[40:35.810 --> 40:39.740]  It monitors behavior within containers.
[40:39.740 --> 40:43.100]  It can detect if a container has been compromised
[40:43.740 --> 40:47.320]  because suddenly the container starts acting weird.
[40:47.320 --> 40:51.920]  Or there are commands being executed inside the container like
[40:51.920 --> 40:55.940]  cat etsy password. Probably not a command
[40:55.940 --> 40:58.980]  that should be showing up in one of your containers.
[40:59.220 --> 41:03.920]  If anything of this entire talk that I'm
[41:03.920 --> 41:07.740]  giving today, you get out of this, go look at falco.
[41:07.740 --> 41:11.280]  This is one of the best tools out there. Kubehunter
[41:11.280 --> 41:15.280]  is another great tool. We're going to talk about that.
[41:15.540 --> 41:19.640]  It's used more so for looking at doing some pen testing.
[41:19.640 --> 41:24.140]  We're going to take a look at that in just a second. Alright, ready, set, attack.
[41:24.140 --> 41:27.700]  If we were going to pen test an environment, this is an
[41:27.700 --> 41:31.520]  example of all the ports and all of the things that are
[41:31.520 --> 41:35.800]  open in a Kubernetes cluster. Now, they're
[41:35.800 --> 41:39.800]  open, but are they open by firewalls and so on?
[41:39.800 --> 41:43.800]  Well, we don't know. The API server
[41:43.800 --> 41:47.680]  itself is, yes, it's HTTPS.
[41:47.680 --> 41:50.140]  It better damn well be protected.
[41:51.720 --> 41:55.920]  etcd on 6666, that should be protected
[41:55.920 --> 41:59.620]  wherever it happens to be. This just gives you an idea. All of these
[41:59.620 --> 42:03.820]  ports can be changed. These are all the defaults in most cases.
[42:04.360 --> 42:07.600]  Just be aware of the ports that are out there and
[42:07.600 --> 42:11.620]  exposed that you need to make
[42:11.620 --> 42:15.720]  sure are protected. Because as an attacker, what am I
[42:15.720 --> 42:20.100]  going to do? Well, one of the first things, even Jason was talking about it in his talk,
[42:20.100 --> 42:24.360]  looking at ports, running nmap, running mass scan, and so on.
[42:25.040 --> 42:27.380]  This is exactly one of the things that
[42:27.380 --> 42:31.700]  Shodan does. It looks for all of these things that are exposed on the
[42:31.700 --> 42:35.700]  internet. So please make sure you're aware of all the ports
[42:35.700 --> 42:39.920]  within your cluster. Now, I are a hacker.
[42:39.920 --> 42:43.720]  See me hack. This is one of the
[42:44.320 --> 42:47.920]  ways that I'm going to start to attack
[42:49.000 --> 42:51.620]  a Kubernetes cluster. I want to first
[42:51.620 --> 42:55.800]  and foremost check access to the API server, make sure that is
[42:55.800 --> 42:59.820]  protected. I'm going to be looking at etcd,
[42:59.820 --> 43:03.920]  and making sure that's protected. If I find it exposed,
[43:03.920 --> 43:06.640]  I'm going to get keys to the kingdom in that case.
[43:06.640 --> 43:11.700]  What about the kubelets that are communicating and the command channels
[43:11.700 --> 43:15.040]  that are there? They should be read-only, but
[43:15.700 --> 43:19.420]  they should not be exposed. There was actually
[43:22.440 --> 43:23.640]  a vulnerability exposed
[43:23.640 --> 43:27.640]  about this, I want to say two months ago. I'm probably off in that
[43:27.640 --> 43:31.920]  sense. But take a look at that. And it allowed, even though it's a
[43:31.920 --> 43:35.740]  read-only port and it's supposed to limit what you can see,
[43:35.740 --> 43:40.500]  there were some vulnerabilities that allowed for a lot of exposure there.
[43:40.500 --> 43:44.160]  Also, container vulnerabilities.
[43:44.160 --> 43:47.800]  What about, you know, do you have files
[43:47.800 --> 43:51.960]  in that container that the application needs that
[43:51.960 --> 43:55.600]  you should be protecting? Well, that's where I would set up Falco and tell
[43:55.600 --> 43:59.440]  Falco, hey, there's a special file over here. This application
[43:59.440 --> 44:03.500]  is using it. But keep an eye on that. Make sure that the only
[44:03.500 --> 44:07.540]  thing that is reading that application is, or excuse me,
[44:07.540 --> 44:10.720]  that is reading that file happens to be the
[44:11.600 --> 44:15.160]  application. And Falco can monitor that.
[44:15.400 --> 44:19.160]  Privilege escalation. Nine times out of ten, once somebody
[44:19.160 --> 44:23.380]  compromises an application in a container, they
[44:23.380 --> 44:27.240]  elevate privilege because there's some other bugs in there.
[44:27.240 --> 44:31.420]  Go back to the Kubotic slide where
[44:31.420 --> 44:35.560]  it showed privilege escalation was not
[44:35.560 --> 44:39.260]  set. All you have to do is set that configuration
[44:39.260 --> 44:43.760]  variable. It's not that hard. Okay?
[44:43.760 --> 44:46.840]  Vulnerable applications are critical.
[44:47.360 --> 44:51.480]  Can't say it enough times. Especially because there are
[44:51.480 --> 44:55.840]  so many people using Kubernetes and Docker to deploy
[44:55.840 --> 44:59.480]  WordPress. And WordPress is, of course, one of
[44:59.480 --> 45:03.500]  our favorite things to deal with. All right. So your
[45:03.500 --> 45:07.320]  final threat model, when it comes to pentesting and
[45:07.320 --> 45:11.560]  looking at the environment that is Kubernetes,
[45:12.220 --> 45:15.400]  you think of your workload. You think of that
[45:15.400 --> 45:19.220]  security within your pods. Can I
[45:19.220 --> 45:23.700]  set up my pods so they're indeed secure?
[45:23.700 --> 45:27.400]  The container security itself, the configuration
[45:27.400 --> 45:31.940]  for that container, the more hooks I put in,
[45:31.940 --> 45:35.620]  run as root, no. Privilege escalation,
[45:35.620 --> 45:39.420]  no. These are the things that we want to make sure that are
[45:39.420 --> 45:43.300]  set. Also, file storage security.
[45:43.300 --> 45:47.440]  Maybe there's a particular file system that has to be read right
[45:47.440 --> 45:51.380]  but the rest of the file system can be read only. Those are
[45:51.380 --> 45:56.500]  other things. The network security. Remember the infrastructure.
[45:56.520 --> 45:59.300]  Put firewalls in. Put controls
[45:59.300 --> 46:03.400]  in there that detect lateral movement. If
[46:03.400 --> 46:07.720]  somebody does compromise one of your applications,
[46:07.720 --> 46:11.360]  make sure you have something monitoring. I gave you that
[46:11.360 --> 46:15.680]  slide way back about the blog entry from Wazoo.
[46:15.680 --> 46:19.420]  They talk about how to audit and monitor
[46:20.000 --> 46:23.440]  a Kubernetes cluster using their tool. I love the Wazoo
[46:23.440 --> 46:27.580]  manager and agent. Take a look at it if you haven't heard of it
[46:27.580 --> 46:31.780]  before. Remember, most
[46:31.780 --> 46:35.500]  breaches are not zero day. I can't say it enough times. Everybody's always
[46:35.500 --> 46:39.600]  trying to protect against this stuff. They're not fancy.
[46:39.600 --> 46:43.380]  Most of these things are being broken into because people
[46:43.380 --> 46:47.540]  make silly mistakes. Because remember, common sense.
[46:47.540 --> 46:51.320]  Whatever happened to it. The breaches are going to come from configuration
[46:51.320 --> 46:55.400]  issues. The kube audit was a very good
[46:55.400 --> 46:59.440]  example where it was showing all the problems that would allow someone to
[46:59.440 --> 47:03.180]  get in. Compromise credentials.
[47:03.180 --> 47:07.260]  Make sure your credentials are stored in a secure place
[47:07.260 --> 47:11.140]  somewhere. And don't allow for
[47:11.140 --> 47:14.880]  privilege escalation. Kubernetes and Docker
[47:14.880 --> 47:18.980]  give you the tools and the configurations to prevent that.
[47:18.980 --> 47:23.320]  Just turn them on. And that's what we're not doing. We have
[47:23.320 --> 47:27.260]  to turn on the configurations. So your key takeaways
[47:27.260 --> 47:31.140]  from today. Common sense.
[47:31.140 --> 47:33.880]  Please go back and stop
[47:34.960 --> 47:40.240]  and think about where is the exposure?
[47:40.240 --> 47:44.660]  Where are my pods exposed? What about my etcd?
[47:45.240 --> 47:48.200]  Configuration. That is so critical to any
[47:48.200 --> 47:52.200]  Kubernetes. And always remember containers.
[47:52.200 --> 47:55.880]  Containers are the linchpin of a Kubernetes
[47:55.880 --> 48:00.140]  environment. And if I don't secure my individual containers
[48:00.140 --> 48:04.320]  then Kubernetes... I don't care about
[48:04.320 --> 48:08.460]  the security of Kubernetes because the container is insecure.
[48:08.460 --> 48:12.720]  So stop overthinking this stuff and think about
[48:12.720 --> 48:17.420]  security basics. Remember the CIS benchmarks.
[48:17.860 --> 48:20.540]  Can't say it enough. Basics.
[48:20.540 --> 48:24.440]  I still think as security professionals
[48:24.440 --> 48:28.480]  hackers, whatever we want to call ourselves. We have
[48:28.480 --> 48:32.340]  forgotten the basics of how to secure an environment.
[48:32.340 --> 48:35.780]  And that's why things keep getting broken into.
[48:35.780 --> 48:40.050]  Stop doing fancy crap. The fancy crap doesn't work.
[48:41.460 --> 48:43.940]  Secure the environment, please.
[48:44.020 --> 48:48.240]  Use the CIS benchmarks for the containers, for Kubernetes,
[48:48.240 --> 48:51.760]  for the OS that your stuff runs on. If you're going to
[48:51.760 --> 48:55.760]  deploy this whole Kubernetes cluster on Ubuntu, then pull up
[48:55.760 --> 48:59.280]  the CIS benchmarks for Ubuntu. Or
[48:59.280 --> 49:04.040]  Red Hat or CentOS. Whatever you're using.
[49:04.040 --> 49:08.060]  Remember, RBAC is your friend. ABAC
[49:08.060 --> 49:12.080]  is not. Your infrastructure better be
[49:12.080 --> 49:15.760]  secure. Your pods better be secure.
[49:15.760 --> 49:20.080]  Containers themselves, the network. Still a lot of people forget
[49:20.080 --> 49:24.160]  the network because they forget about TLS everywhere.
[49:26.880 --> 49:28.300]  Threat model
[49:28.300 --> 49:32.000]  your applications. If you don't take
[49:32.000 --> 49:36.500]  the beginning effort to threat model your applications,
[49:36.500 --> 49:40.180]  you end up with insecure applications. And all the security
[49:40.180 --> 49:43.580]  of your Kubernetes cluster is useless because you have
[49:43.580 --> 49:48.260]  insecure applications within your environment. Access controls
[49:48.260 --> 49:51.980]  go along with RBAC, but access controls into the
[49:51.980 --> 49:56.460]  environment, kind of referencing SSH.
[49:56.920 --> 49:59.760]  I found one worker node configuration
[49:59.760 --> 50:03.960]  in an environment probably nine months ago. But for some
[50:03.960 --> 50:08.700]  reason on the worker nodes, they had FTP enabled on all the worker nodes.
[50:08.820 --> 50:12.660]  And my head exploded.
[50:13.180 --> 50:16.540]  It allowed me to get into everything, but oh well.
[50:17.380 --> 50:19.920]  Yes, did I mention RBAC though?
[50:20.220 --> 50:23.480]  Enough times. I can't say it enough. Why? Because
[50:23.480 --> 50:28.000]  people tend to ignore it. Remember the tools I mentioned earlier
[50:28.000 --> 50:32.140]  about monitoring or auditing the RBAC roles and also
[50:32.140 --> 50:35.960]  the permissions. And finally,
[50:35.960 --> 50:39.900]  not quite finally, remember to patch everything.
[50:39.900 --> 50:43.820]  The infrastructure that goes along with that.
[50:43.860 --> 50:48.080]  Logging and auditing and monitoring. I still see people
[50:48.080 --> 50:52.700]  set up their environments. There's that blog entry again.
[50:52.700 --> 50:56.040]  Please take a look at that. It is well written.
[50:56.040 --> 50:59.960]  It's a great tool. You'll learn a lot about Wazoo. I use it on
[50:59.960 --> 51:04.120]  all my systems. I deploy a lot of honeypots with
[51:04.120 --> 51:08.100]  Kubernetes and containers and things like that. But I monitor them all
[51:08.100 --> 51:12.240]  with Wazoo. You can't
[51:12.240 --> 51:16.540]  have a secure environment if you're not monitoring that environment.
[51:16.540 --> 51:18.360]  That is the biggest thing.
[51:19.860 --> 51:24.080]  That brings me to the end. I thank you for your
[51:24.080 --> 51:28.180]  attention. There's contact info. There's my Twitter
[51:28.180 --> 51:32.260]  account. I will get this to the Red Team
[51:32.260 --> 51:36.360]  Village and they can upload it to their environment. But I'll have it on my
[51:39.160 --> 51:40.200]  GitHub later
[51:40.200 --> 51:44.360]  in an hour or so. I'll update it there. Thank you all very much.
[51:44.360 --> 51:48.140]  Hopefully you learned something today. Go out and hack and have some fun. Thanks to
[51:48.140 --> 51:51.480]  the Red Team Village and have an awesome DEF CON.
[51:51.480 --> 51:55.840]  Awesome. Thank you so much, Kat. Amazing presentation
[51:55.840 --> 51:59.500]  as always. Thank you for supporting DEF CON. Thank you for supporting the
[51:59.500 --> 52:02.700]  community. Of course, the Red Team Village. For those of you
[52:03.440 --> 52:06.940]  that just joined or are looking at
[52:07.340 --> 52:11.200]  the presentation on Twitch or YouTube, please look at
[52:11.200 --> 52:15.480]  all the activities that we have going on. Go to the website that is in the description of
[52:15.480 --> 52:19.540]  the stream. Please also join the conversation in Discord.
[52:19.540 --> 52:23.760]  We're going to go on a brief break and the next presentation will start in a few minutes.
[52:23.760 --> 52:25.340]  Thank you again and thank you, Kat.
